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On the surface, it appears that AS9100 1 has little to say about how to apply a Quality 
Management System (QMS) to major aerospace test programs (or even smaller ones). It also 
appears that there is little in the quality engineering Body of Knowledge (BOK) that applies to 
testing, unless it is nondestructive examination (NDE), or some type of lab or bench testing 
associated with the manufacturing process. However, if one examines: a) how the systems 
engineering (SE) processes are implemented throughout a test program; and b) how these SE 
processes can be mapped to the requirements of AS9100, a number of areas for involvement of 
the quality professional are revealed. What often happens is that quality assurance during a test 
program is limited to inspections of the test article; what could be considered a manufacturing al 
fresco approach. This limits the quality professional and is a disservice to the programs and 
projects, since there are a number of ways that quality can enhance critical processes, and 
support efforts to improve risk reduction, efficiency and effectiveness. 

The Systems Engineering (SE) discipline is widely used in aerospace to ensure the 
progress from Stakeholder Expectations (the President, Congress, the taxpayers) to a successful, 
delivered product or service. Although this is well known, what is not well known is that these 
same SE processes are implemented in varying complexity, to prepare for and implement test 
projects that support research, development, verification & validation, qualification, and 
acceptance test projects. Although the test organization's terminology may vary from the SE 
tenninology, and from one test service provider to another, the basic process is followed by 
successful, reliable testing organizations. 


For this analysis, NASA Procedural Requirements (NPR) 7123.1, NASA Systems 
Engineering Processes and Requirements is used to illustrate the SE processes that are used for 
major aerospace testing. Many of these processes are also implemented for smaller test projects, 
and this set of processes will also look familiar to those who have participated in launch site 
activation and flight demonstrations. 


1 SAE AS9100 Quality Management Systems - Requirements for Aviation, Space and Defense 
Organizations, 
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‘ http://cert.asq.org/certification/control/quality-engineer/bok 


3 http://nodis3.gsfc.nasa.gov/displayDir.cfm?t=NPR&c=7123&s=TB 
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Figure 1 NPR 7123.1 Systems Engineering "Engine" 
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In studying the Seventeen Processes of the SE "Engine", quality assurance professionals 
will find many opportunities to apply the lessons from Deming 4 and Crosby 5 for process 
improvement and defect prevention (e.g. planning, design, continual improvement, supplier and 
process capability assessments, process control, education and training), and will leam that 
quality assurance as related to testing is not limited to quality control inspections after the test 
article is delivered to the test site. It is important to note also that many project managers and 
systems engineers, acting as test customers, are not fully aware of the complex systems 
engineering processes required to fulfill their agreements with testing organizations. This can 
result in increased technical and project risk. 

The complexity of the processes will depend upon the life cycle phase (e.g., research vs. 
qualification), the level of the test article and/or software (e.g., component vs. system), the type 
of test (e.g., subscale wind tunnel test vs. integrated stage hot fire), and other factors such as the 
use of heritage hardware, or repeated testing at the same test location. It is important to note that 


4 http://asq.org/learn-about-quality/total-quality-management/overview/deming-points.html 

5 http://asq.org/learn-about-quality/cost-of-quality/overview/overview.html 
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these processes also take place at launch sites, but for simplicity, this analysis will use the word 
“test”. 


The graphic shown above from NASA’s NPR7 123.1, is used for the following 
discussion. Processes 1 and 2, Stakeholder Expectation Definition and Technical Requirements 
Definition, respectively, are mainly performed at the project level (i.e., identification of mission, 
integrated systems, systems, subsystems, components) before the testing organization is 
contacted. However, information at this point is often preliminary, and the testing organization 
may participate in this process in parallel with Technical Planning (Process 10) very early in the 
life-cycle, especially if new facilities and/or hardware are needed. In the interest of maintaining a 
schedule, this entire process can be iterative, often with final requirements and as-built drawings 
being completed in time for the Test Readiness Review, which is a type of Technical Assessment 
(Process 16). Therefore, each organization needs some level of work tracking and constraint 
checks to confirm readiness (i.e. Configuration and Data Management (Processes 14 and 15). 

The Test Team must go through the logical decomposition process (Process 3) with the 
customer’s Test Requirements Document in order to thoroughly understand the required 
parameters and derived requirements. They must begin the process of identifying test 
equipment, test facilities, support equipment, fixtures, software needs, and instrumentation. This 
is closely followed by design of equipment, fixtures, new facilities or modifications, data 
channels, and coding of software (Process 4). 

The Test Team and the customer must engage in Technical Planning (10) and other 
Technical Control Processes (11-15) throughout this activity. Depending on the magnitude of the 
effort, this could be an iterative process that could take many months. Technical requirements, 
trade studies, cost, schedule, safety, quality assurance, enviromnental regulations, logistics, 
calibration, procurements, workforce staffing and training, permits and certifications, codes and 
standards must be identified, managed (Process 1 1), implemented and tracked. Requirements 
changes from the customer must be factored in and schedule adjustment may be necessary, 
depending upon the timing and magnitude. Interface Management (12) is part of this overall 
planning and control process, since test article-to-equipment interfaces must be identified and 
tracked, as well as each interface at the facility, equipment, and fixture level. For purposes of 
mapping to AS9100, this relates to Configuration Management. Depending on the complexity of 
the test and test article, some type of Interface Control Document may be required. 

Technical Risk Management (Process 13) can also be a significant aspect of preparations, 
again, depending upon the magnitude of the effort. Much of this activity proceeds at some level 
of risk, due to the fact that many customer requirements are still preliminary. Consequently, 
facility and equipment fabrication may have to proceed with preliminary drawings, in some 
cases. It is often accepted that the schedule risk of changes to test article and/or test 
requirements, resulting in redesign of the facility or equipment, has a high probability, but lower 
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consequences, than the cost and schedule risk of waiting until later in the life cycle to begin the 
process. 

Configuration of facilities, fixtures, software, and equipment and of the work authorizing 
documentation must be carefully managed (Process 14) and all associated documents, drawings, 
reports, calculations, analyses, and test data must be managed and maintained in an accessible 
records system (Process 15). This is vital to the Technical Assessment (i.e. test readiness review, 
hazard analysis) Process (16), and for use of the data by the customer. 

At some point during the early and later planning stages, purchases are made and 
products delivered, such as off-the-shelf valves, tubing, equipment, instrumentation, computers, 
and possibly liquid propellants or pressurizing gases. Test organizations often have the ability to 
fabricate facility modifications, support equipment, and test equipment. This is Product 
Implementation for the test organization, in order to be configured for a specific test article 
(Process 5). It should be noted that the industry codes and standards, which provide the safety 
and reliability of the test facilities and equipment, are based on very exacting industry quality 
assurance methods and controls. There may also be support operations such as precision cleaning 
and calibration laboratories, which must meet exacting standards. Test software must be coded 
and configuration controlled. Test facility hardware and software must be installed and 
integrated (Process 6). This process will eventually include integration of the test article with the 
test facility or equipment. This includes hardware, control software, and instrumentation. 

Verification (Process 7) and Validation (Process 8) (“V&V”) are also required for the 
test facility, equipment and software. For safe and reliable operations, all new or modified 
systems, along with existing systems, must be “baselined” through some type of inspection and 
analysis process, and tested to detennine whether the systems are perfonning as expected and 
required, including both planned commands and emergency shutdown. Additional V&V 
activities may be required after the test article is integrated into the facility system. This could 
include various types of checkouts, such as proof testing, cold flows, ignition testing, 
pathfinders, engineering units and calibration samples and sequence runs. There may be some 
type of Operational Readiness Review (Process 16) in parallel with this design, integration and 
V&V process that is required by the test organization separately from the customer's processes 
(although a project review can be done in conjunction with a facility review). 

All of this leads up to the Test Readiness Review (Process 16), at which time a board 
is convened to assess readiness of facility, test article, and crew; safety of personnel and 
operations; risk, adequate level of quality assurance, environmental compliance, emergency 
shutdown capabilities, and other pertinent topics. Once pennission to proceed with testing is 
granted, the V&V of the test article can begin, based on the customer's requirements. Depending 
on the test, many months of planning, fabricating, purchasing, integrating and facility V&V must 
take place before this point. 
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The test article is transitioned back to the test customer once the test or test series is 
complete (Process 9) and the Test Team may participate in data reviews to support the Decision 
Analysis (Process 17) that the customer must perform. Product Transition, in reality, is the 
delivery of the test data, which is provided in some pre-determined electronic format, along with 
any required documentation. The testing organization is responsible for the integrity of the test 
data, while validation of the test article’s perfonnance is the responsibility of the customer. 

Some extrapolation is required to apply AS9100 to a testing lab, since testing is a 
unique combination of building and maintaining infrastructure, and providing a service that 
requires this infrastructure. However, once the foundation is laid for understanding the 
implementation of SE processes in a testing environment, the next step is to map the SE 
processes to AS9100, as shown in the table below. Once the relationship between AS9100 and 
SE, and between SE and test organization processes becomes clearer, it can be seen how a QMS 
can be developed and implemented that serves the testing organization, rather than forcing a 
product manufacturing paradigm onto a testing service. This understanding is vital, since 
shortcuts in quality assurance may actually be shortcuts in the SE processes. 

One final caveat for the auditors: The measure of the effectiveness of a testing 
organization is found in the test data. It is the data that reveals whether the customer's 
requirements have been met, for example, by the test sequence, in recorded temperatures, 
pressures, flow rates, durations, loads, acoustics, and many other types of inputs. Test procedures 
are often the necessary steps to prepare for the test and to condition the test article. The test itself 
may in fact be run by the test control computer and recorded by instrumentation, often on several 
hundred data channels at several hundred samples per second. The test procedure may not be the 
build-paper equivalent that the auditors are looking for, since much of it is often facility related. 
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Table 1 Systems Engineering Processes Mapped to AS9100 


SE PROCESS 

AS9100 REQUIREMENT 

Stakeholder Expectations 

Cu.vtomer Requirements 

Technical Requirements 
Definition 

Planning of Product Realization 

Logical Decomposition 

Design and Development Input 

Design Solution Definition 

Design and Development Output 

Product Implementation 

Control of Production 

Product Integration 

Control of Production 

Product Verification 

Verification 

Product Validation 

Validation 

Product T ran sit ion 

Control of Work Transfers; Post Delivers Support, 
Preservation of Product 

Technical Planning 

Planning of Product Realization; Review of Requirements; 
Measurement, Analysis and Improvement 

Requirements 

Management 

Design and Development Planning; Purchasing 

Interface Management 

Configuration Management 

Technical Risk 
Management 

Risk Management 

Configuration 

Management 

Configuration Management; Identification and Traceability; 
Control of Nonconforming Product 

Technical Data 
Management 

Control of Documents; Control of Records; Control of Design 
and Development Changes 

t echnical Assessment 

Design and Development Review 

Decision Analysis 

Measurement, Analysis and Improvement; Analysis of Data 


As can be seen, "test" can entail a significant SE effort in the life cycle of a product, 
which can be underestimated by project managers and overlooked by quality professionals. The 
application of AS9100 to test activities should not be limited to acceptance and qualification of 
the test article, since a QMS can be mapped to SE processes that are implemented during 
preparations for testing at any point in the life cycle. The aerospace quality professional needs to 
be able to relate AS9100 to systems engineering and to aerospace testing, so that an approach to 
quality assurance involvement beyond test article inspection can be formulated to achieve 
performance excellence. 




























